Scheduling system for transmission of cells to ATM virtual circuits and DSL ports

ABSTRACT

A system and method for controlling transmission of cells is described. The cells are associated with virtual circuits that either require shaping according to constant bit rate (CBR) or real-time variable bit rate (rt-VBR), or no shaping with transmit selection based on priority (for services other than CBR and rt-VBR). The system transmits the shaped and unshaped traffic using one or more circular control structures. The control structures have time slots at the granularity of the maximum system transmit rate.

BACKGROUND

[0001] The invention relates generally to networking, and moreparticularly to cell-based transmission scheduling for virtual circuits.

[0002] In complex networks, cells are transmitted to physical channels(or “ports”) over a virtual circuit according to traffic parameters ofthe virtual circuit. One example of such traffic parameters are thosespecified in the “The ATM Forum Technical Committee Traffic ManagementSpecification Version 4.1”, The ATM Forum, March 1999. An ATM virtualcircuit connection may characterize its traffic by using source trafficdescriptions, which attempt to capture the cell inter-arrival patternfor resource allocation. Once such traffic descriptor is Peak Cell Rate(PCR), which represents the minimum spacing between cells, and thereforethe peak emission rate of the source. The PCR is expressed incell/seconds.

[0003] ATM supports a quality of service required by an applicationthrough the selection of an appropriate service category. The servicesoffer different QoS commitments in terms of delay and loss tolerance.The services differ in how the network allocates bandwidth and appliesdifferent traffic management functions. The service categories includeconstant bit rate (CBR), variable bit rate (VBR) and unspecified bitrate (UBR). For CBR and VBR, bandwidth is allocated for the duration ofthe connection. In contrast, UBR services target for use bandwidth thatbecomes dynamically available as connections go idle.

[0004] The CBR service provides a connection with dedicated bandwidthproviding extremely low probability of cell loss, as well as low andpredictable delay. The inter-arrival time between two cells is constantand can be characterized as a minimum cell inter-arrival, whichcorresponds to a known PCR.

[0005] The VBR service category is mainly intended for more efficientsupport of applications that have known or predictable bursty trafficcharacteristics. The VBR traffic can be characterized by a sustainedcell rate (SCR) as well as a PCR. The SCR is measured over a definedperiod and represents the average transmission rate. The VBR service canbe further divided into two subcategories based on delay requirements,the real-time VBR (rt-VBR) and non-real-time VBR (nrt-VBR). The rt-VBRhas strict end-to-end delay requirements, whereas the nrt-VBR does notguarantee any delay bounds.

[0006] A UBR virtual circuit can have a priority determined by theweight associated with such VC on a given port.

[0007] Ports can be characterized in terms of the maximum rates at whichthey are capable of transmitting.

DESCRIPTION OF DRAWINGS

[0008]FIG. 1 is block diagram of a network environment that includes aDigital Subscriber Loop Access Multiplexer (DSLAM).

[0009]FIG. 2 is a block diagram of DSLAM, including a Segmentation andRe-assembly device (“SAR”).

[0010]FIG. 3 is a block diagram illustrating one embodiment of the DSLAMbased on a network processor architecture.

[0011]FIG. 4 is a detailed block diagram of the SAR device (of FIG. 2).

[0012]FIG. 5 is a depiction of the field format of an entry in acircular buffer that maintains schedules for shaped traffic (“shapedwheel”).

[0013]FIG. 6 is a flow diagram of a receive process used by SAR device.

[0014]FIG. 7 is a flow diagram of a scheduler process used by the SARdevice.

[0015]FIGS. 8A and 8B are flow diagrams of schedule determinationprocesses for shaped traffic and unshaped traffic, respectively.

[0016]FIG. 9 is a depiction of exemplary circular buffers (or “wheels”)for shaped and unshaped traffic.

[0017]FIG. 10 is a depiction of an exemplary port table.

[0018]FIG. 11 is a flow diagram of a traffic shaping process used by theSAR device.

[0019]FIG. 12 is an exemplary multi-threaded microengine embodiment ofthe SAR traffic scheduler and shaping functions.

[0020]FIG. 13 is a depiction of the multi-threaded microengineembodiment of the receive, scheduler and traffic shaping functions inwhich the shaped and unshaped wheels each are partitioned into fourseparate wheels.

[0021]FIG. 14 is a detailed block diagram of an alternative embodimentof the SAR device (of FIG. 2).

[0022]FIG. 15 is a depiction of the field format of an entry in acircular buffer that maintains schedules for shaped and unshaped trafficin the SAR device of FIG. 14.

[0023]FIG. 16 is a flow diagram of a scheduler process used by the SARdevice of FIG. 14.

[0024]FIGS. 17A and 17B are flow diagrams of schedule determinationprocesses for shaped and unshaped traffic, respectively, performed bythe scheduler of FIG. 16.

[0025]FIG. 18 is a flow diagram of a traffic shaping process used by theSAR device of FIG. 14.

DETAILED DESCRIPTION

[0026]FIG. 1 shows a DSL network environment 10 that includes a DSLaggregation device 12, shown as a Digital Subscriber Loop AccessMultiplexer (DSLAM), which concentrates connections 14 a, 14 b, . . . ,14 k, from DSL access points 16 a, 16 b, . . . , 16 k, for access to aservice network such as the public Internet (or a corporate Intranet)18. The DSL network environment 10 may be viewed in two parts: acustomer or service user environment 20 and a service providerenvironment 22. In the customer environment 20, the DSL access points 16typically correspond to Customer Premises Equipment (CPE). The CPE cantake a variety of different forms, e.g., a DSL modem used by a homeconsumer, or a Small Office/Home Office (SOHO) router, and so forth. Theconnections 14 between the CPE 16 and the DSLAM 12 are usually ATMconnections. The DSLAM 12 can be deployed in the service providerenvironment 22, as shown.

[0027] The DSLAM 12 can be characterized as having a CPE side with firstport interfaces 24 for handling ATM cell-based traffic associated withcorresponding DSL links or connections 14, and one or more second portinterfaces 26, which are coupled to a router (or ATM switch) 28 via aWAN uplink connection 30. The router/switch 28 connects to a servicenetwork, such as the Internet 18, as indicated earlier, or some othertype of service network, for example, an ATM network 32. Thus, forupstream traffic, many DSL ports on the CPE side may be aggregated atthe DSLAM 12 and, on the service provider side, connected to the servicenetwork router with a single physical port interface.

[0028] For each port, there may be many virtual connections. The virtualconnections represent “state full” communication setups, such as an ATMvirtual circuit or Internet TCP connection. At each end of the networkvirtual connection is an application that can send and receive messages.The messages are carried across the network as packets or frames, whichare further subdivided into 48 byte ATM cells. The interface into andout of the DSLAM 12 is either cell-based (48 byte ATM cells) or packet(or segment) -based (64 byte packet or greater sized segments based onthe MTU on port interfaces 26). In the embodiment shown, the first portinterfaces 24 are cell-based and the second port interfaces 26 handleframes (or packets). Each virtual connection has a quality of service orrate specification. In the described embodiment, the types of ratesinclude constant bit rate (CBR), real-time and non-real-time variablebit rate (rt-VBR and nrt-VBR, respectively) and unspecified bit rate(UBR). A priority may be associated with a VC that contracts with theUBR service.

[0029] Referring to FIG. 2, a depiction of the DSLAM 12 for handlingtraffic from the service network 18 (or 32) to one of the CPEs 16 (thatis, traffic flowing in the downstream direction) is shown. The DSLAM 12therefore includes at least one of the second port interfaces 26, shownas an ingress medium interface, and at least one of the first portinterfaces 24, shown as an egress medium interface. The interface 26receives the downstream traffic from a service network, such as network18, and provides that packet-based traffic to a Segmentation andReassembly unit (SAR) 34. The SAR 24 segments packets into ATM cells,which are transmitted to a CPE over a medium via the egress mediuminterface 24. The SAR 34 also performs traffic scheduling and shaping,as will be described. The DSLAM 12 could further include logic toperform optional pre-SAR processing (e.g., packet formatting,aggregation of and interface to different media interfaces 26) andpost-SAR processing (e.g., interface and MUX to different mediainterfaces 24). The logic could be implemented in software or hardware,e.g., in Field Programmable Gate Arrays (FPGAs) 36, 38, as shown. Thus,the logic 36 and 38 removes from the SAR 34 any complexity related tospecific media interfaces and connects to the SAR 34 with a single busthat is compatible with the SAR architecture. The bus communicationbetween the SAR and external processing logic 36, 38 is therefore purelyconcerned with satisfying the particular handshaking signals required bythat bus.

[0030] The SAR 34 may be implemented with a commercially availablenetwork processor, for example, the Intel® IXP™ 1200 network processor.In such an embodiment, as shown in FIG. 3, the SAR 34 may be coupled toeach FPGA via the Interface Exchange (IX) Bus. In this or other networkprocessor architecture implementations, the bus could be some other bussuitable for connecting the SAR 34 to a medium interface or optionalprocessing logic. Specifically, FIG. 3 illustrates the SAR 34, alongwith the DSLAM unit's pre-SAR cell-based processing FPGA 38 and aplurality of ingress medium interfaces 26. The interfaces 26 can beimplemented to handle different types of connectivity and medium accessprotocols.

[0031] Referring now to FIG. 4, the SAR 34 includes processes andcontrol structures for a concurrent shaped and unshaped trafficscheduling system. It will be appreciated that the process and controlstructures illustrated in the figure are only those that pertain toprocessing of traffic flowing in the downstream direction, that is, fromservice network to service user. All of these processes and structuresreside on, and the processes are executed by, a processor such as anetwork processor, as discussed above with reference to FIG. 3. Theprocesses include a receive process 50, a scheduler process 52 and ashaper/transmit process 54. The control structures include receivecontrol structures 56, shaper/transmit control structures 58 and ascheduler control structure 59. The receive control structures 56include the following: a VC table 60 and VC packet queues 62. The VCtable 60 stores information associated with different VCs, such aspre-built ATM header, traffic parameters and state information (queuedepth, thresholds, etc.) associated with VC queue and port destinationinformation. The lookup table 60 is indexed by values that correspond toVCI/VPI information or, alternatively, for a table of reduced size,indexed by values generated by hashing VPI/VCI information. The VCpacket queues 62 store incoming packets by the VCs associated with thepackets. Thus, the VC packet queues 62 are organized by an identifier(such as the VCI) by which the associated VC can be determined. Theindex is used to determine the appropriate VC (or flow) entry in the VCtable 60 to be read and the specific queue in which the incoming packetis to be stored.

[0032] The port specific control structures 58 include ports tables 66,an array of UBR VCs 67, a “first chance” queue 68 and a Queue WithPacket (QWP) vector 70. The scheduler control structures 59 include ascheduler message queue 72, as well as a shaped VC control structure 74and an unshaped VC control structure 75. In the illustrated embodiment,the VC control structures 74 and 75 are implemented as calendarqueues,and are thus depicted conceptually and referred to hereinafter as“wheels”.

[0033] The scheduler message queue 72 stores schedule requests generatedby the receive process 50 or shaper process 54, and is read by thescheduler process 52. The requests are cell transmission schedulingrequests, which are VC-specific, and are therefore associated with oneof the service types, that is, CBR, rt-VBR, nrt-VBR or UBR. The shapedVC wheel 74 and the unshaped VC wheel 75 hold current schedules. Morespecifically, the shaped VC wheel 74 is used to schedule CBR and rt-VBRVCs and, at times, nrt-VBR VCs (to meet SCR) as well, whereas theunshaped VC wheel 75 is used to schedule nrt-VBRVCs (with theaforementioned exception) and UBR VCs and, at times (e.g., toaccommodate schedules to be added when a port goes from a flow-controlasserted to de-asserted state), CBR and rt-VBR VCs as well, as will beexplained in further detail later. The wheels 74, 75 are implemented asan array of time slots, the wheel 74 including slots 76 and the wheel 75including slots 77, and the slots 76, 77 correspond to the maximumtransmit rate of the network processor. Each slot represents a time slotin which an ATM cell can be transmitted.

[0034] The number of slots in the wheels is a function of aggregate portbandwidth. For example, for an aggregate bandwidth of 622 Mb/s, say, thesmallest granularity of bandwidth that can be supported on any port is9.6 Kb, which means that the wheels would be partitioned into 64 klocations or slots.

[0035] Cell transmit rate refers to the spacing between cell transmitsto the network by the shaper/transmit process 54. The shaped wheel 74 isoperated in absolute time. It steps every n cycles regardless of whetherthere are any cells to be transmitted. The timing of the unshaped wheel76 is relative (with a current shaped wheel slot index), as will bedescribed. It advances only when the shaped wheel has no cell to betransmitted in the current time slot.

[0036] A VC index associated with a virtual connection is stored in aparticular one of the slots 76, 77, during scheduling. The VC may have atraffic parameter that requires it to be serviced at least at apredetermined rate. For example, a CBR VC must conform to a PCRspecified for that VC. An rt-VBR must conform to PCR, SCR and MaximumBurst Size (MBS) parameters specified for that VC. Both CBR and rt-VBRwill need to conform to the maximum cell transfer delay (maxCTD) as ameasure of service quality. Cell Delay Variable Tolerance (CDVT) is usedas a measure of time to compensate for jitter introduced due to thescheduling inefficiency.

[0037] The ports have data rates that can be measured and constrained.Like the PCR/SCR rate (bandwidth) associated with VCs, the rate of aport is converted from bits/sec to number of time slots determined bythe port's inter-cell gap: more time slots yield a smaller rate. Theconversion takes into account the network processors clock frequency andthe clock cycle budget to process a single cell based on the aggregatebandwidth (sum of bandwidth of all ports).

[0038] One constraint consideration when configuring ports is the ratesof VCs allocated to a port. Allocation is limited such that the sum ofthe minimum service rates for VC does not exceed the desired rate of theport. This ensures that all VCs on a given port can be serviced to theirminimum rates.

[0039] The VC is scheduled into a wheel with sufficient spacing fortransmission with sufficiency frequency to ensure that the transmissionrate conforms to both the VC cell transmit rate and the port transmitrate.

[0040] The QWP vector 70 is a bit vector in which one bit positioncorresponds to the first chance queue 68 and 16 bit positions correspondto UBR VCs associated with a port. The array of UBR VCs 67 includes upto 16 entries, and each entry's priority is determined by the weightassociated with it. Each array element is associated with a VC by avalue indicating the VC index of that VC and contains the Weighted RoundRobin (WRR) parameters.

[0041] If a bit position in the QWP vector has a value of one, eitherthe first-chance queue or one of the 16 VCs has data awaitingtransmission. When queuing packets in the VC queues associated with a VCthat is either a UBR or a nrt-VBR VC that does not conform to SCR, thereceive process 50 sets the corresponding bit in the QWP vector to one.When the transmit process 54 empties the VC queue queue (correspondingto the VBR-nrt or UBR), it sets the corresponding bit to zero.

[0042] The per-VC packet queues 62 include linked list queues, eachassociated with some VC. Given a VC index, the transmit process 54 cango to the associated per-VC packet queue 62 to get packet descriptorinformation and locate the corresponding packets. The transmit process54 performs segmentation (that is, segments the packets into ATM cells)according to well-known SAR techniques.

[0043] As indicated earlier, the wheel slots represent processor timeslots. Shaped wheel slots 76 reference VCs having “shaped” service ratesCBR and rt-VBR, or sometimes SCR-conforming nrt-VBR. The allocation ofthe slots 76 to VCs provides a schedule for regular service to thoseVCs. Unshaped wheel slots 77 typically reference VCs having “unshaped”service rates nrt-VBR (PCR conforming) or UBR, but, as mentioned above,may also reference VCs having CBR and rt-VBR service rates.

[0044] The slots 76 of the shaped wheel 74 each reference VCs from twogrades of service rates. FIG. 5 illustrates the format of the shapedwheel slot 76, which includes a must-send VC index field 78corresponding to a first service rate (“must send” service rate) gradeand a could-send VC index field 79 corresponding to a second servicerate (“could send” service rate) grade. These fields 78, 79 referencevirtual circuits characterized by the “must send” and “could send”grades of service rate, respectively. A “must-send” grade indicates VCssatisfying PCR for CBR VCs or SCR for VBR VCs. The “could-send” gradeindicates VCs being opportunistic as in the case of rt-VBR VCssatisfying PCR, but of a lower priority than must-send.

[0045] In filling slots in the wheels, the scheduler 52 must ensure thatthe VC cell rate obeys the VC traffic parameters but does not exceed thephysical port rate, as will be described later with reference to FIGS. 7and 8A-8B.

[0046] Virtual circuits are selected for transmission from the unshapedwheel 76 when must-send and could-send virtual VCs in the shaped wheel,which have a higher priority, have not been scheduled. Thus, theunshaped wheel 75 provides rate control for VCs that are prioritizedbehind VCs referenced by must-send 78 and could-send fields 79 on theshaped wheel slots.

[0047] Returning to FIG. 4, the port tables 66 each contain informationused by the scheduler 52 in determining schedules. Each port table 66contains entries for a given port. Each schedule type (must-send,could-send, unshaped) has a separate field that indicates when a VC ofthat type was last scheduled on that port. This ensures that VCsassociated with different schedule types can be scheduled in a way thatdoes not allow a VC of an unshaped schedule type to be scheduled aheadof a VC of a shaped schedule.

[0048] The first chance queue 68 stores references to VCs, in FIFOorder. First chance queue 68 is used for traffic to be transmitted to aVC at the first opportunity, as will be described.

[0049] The shaping/transmit process 54 is a process that manages thecontention of multiple VCs, having various service rates, fortransmission. The shaping/transmit process 54 iterates over the slots ofthe wheels to select VCs for transmission. In general, theshaping/transmit process 54 determines how time slots associated withthe wheels 74, 75 will be used to transmit cells.

[0050] Further details of these control structures will be provided inthe description of the operation of the SAR downstream processes 50, 52and 54 to follow.

[0051] Incoming packets arrive and are stored in a receive buffer inmain memory (not shown) pending transmission. Receive process 50validate cells from the receive buffers and stages them by enqueuing thepackets in the VC packet queues 62, pending transmission by the transmitprocess 54. The receive process 50 enqueues the packets based on the VCindex of the VC with which each packet is associated. The transmitprocess 54 dequeues the packets from the VC packet queues 62, segmentsthe packets into cells and the transmits cells at specified cell ratesappropriate to the VC and destination port, as will be explained. The VCindex gives the position of the VC in VC table 60.

[0052] Referring to FIG. 6, the receive process 50 receives 80 adownstream packet from the network 18 (via one of the interfaces 24 andlogic 36, as discussed earlier). The receive process 50 determines 81 ifthe packet is a control packet or a data packet. If the packet is acontrol packet, the receive process 50 checks 82 for a change in portflow control status. If no change is detected, the receive process 50goes to sleep (at 83). If, at 81, the packet is determined to be a datapacket, the receive process 50 performs a lookup 84 in the VC lookuptable 60 for the received packet to associate that packet with a VC anduses the VC to locate a corresponding one of the packet queues.Information about the VC, e.g., TM4.1 traffic management, current packetbeing assembled, etc., is kept at the table 60, in a specific VC entry.Alternatively, packet data can be received for other network protocols,such as Ethernet/IP, Frame Relay, etc., and a lookup is performed on theheader to obtain an index to the VC in VC table 60. Once the VC and acorresponding packet queue are identified, the receive process 50determines 85 if a fullness threshold is exceeded for the VC packetqueue. If so, the process 50 asserts 86 a flow control signal to theFPGA 36 to push the congestion towards the upstream edge of the DSLAM ormove it outside the DSLAM. Otherwise, the process 50 enqueues 88 thepacket in the VC packet queue. The process 50 determines 90 if the VCpacket queue, prior to receiving the current packet, had been empty. Ifit is determined that the VC packet queue had been empty, or a change inthe port flow control status is detected at 82, the receive process 50sends 92 a schedule request message to the scheduler 52. If the VCpacket queue had not been empty prior to the queuing of the currentpacket, then the process continues processing incoming packets at 80.

[0053] In the described embodiment, to send a message to the scheduler52, the receive process 50 stores a schedule request in the messagequeue 72. This is done only when the VC packet queue transitions from anempty to non-empty state as a result of a new packet being stored inthat VC packet queue, as discussed above. The scheduler 54 does not lookfor work in the VC packet queue. It waits to be told (via the enqueuingof a scheduler request) that there is work to do for a particular VC.This is a work conserving property of the scheduler 54. The messagequeue 72 is also used for any work-related messages by the shaper 54 forthe scheduler 52.

[0054] In particular, when queuing packets for CBR or VBR or UBR, thereceive process 50 sends a message to the message queue 72 to requesteither a shaped scheduling (CBR, rt-VBR) or unshaped scheduling (UBR,nrt-VBR) for transmission.

[0055] Referring to FIG. 7, the scheduler process 52 retrieves 100 fromthe message queue 72 a next scheduling request in the message queue 72.The scheduler process 52 determines 102 if the request is for a “shaped”VC, that is, for a VC having a traffic parameter that requires trafficshaping, such as CBR or rt-VBR (or even nrt-VBR, in order to meet an SCRcontract). If so, the scheduler process 52 clears 104 the previousschedule if the message originated from the shaper 54. If the VC isshaped and the message is from the receive process, the schedulerprocess 52 reads 106 traffic parameters (from the VC table 60) andcomputes a schedule for a next cell on the VC in the shaped wheel. Thescheduler 52 sets 108 the schedule in the shaped wheel. The scheduler 52updates 109 the appropriate port table and VC table with the most recentscheduling information and returns to 100 to process the next receivedmessage.

[0056] If, at 102, the VC is determined to be an unshaped VC, and thescheduler 52 determines 110 that the first chance queue is not empty,the scheduler 52 de-queues 112 an entry from the first chance queue. If,at 110, the scheduler determines that the first chance queue is empty,the scheduler determines an unshaped VC to be processed based on a WRRweight 114. The first chance queue is checked on packet boundaries (asopposed to cell boundaries).

[0057] The simple case of the WRR is the round robin. If there are Nconnections, each separately queued, the RR mechanism in each cyclevisits each of the queues and servers a cell if any is waiting. Thus,the RR mechanism shares the link bandwidth equally among all of thequeues. Instead of an equal share, a weighted share per queue is alsopossible by assigning weights to each queue and giving slotsproportional to the weight in each cycle.

[0058] Thus, when the scheduler 52 considers virtual circuits of annrt-VBR or a UBR type, it uses the Queue With Packets bit vector 70 tocheck if the bit corresponding to the first-chance queue is set. If itis not set, the scheduler 52 uses the WRR algorithm to find the next UBRVC in the array of UBR VCs 67 that has data to transmit.

[0059] The scheduler determines 116 the schedule for the next cell onthis VC in the unshaped wheel. The scheduler 52 sets 118 the schedule inthe unshaped wheel by storing the VC index in the appropriate slot. Thescheduler 52 updates 109 the appropriate port table and VC table withthe most recent scheduling information and returns to 100 to process thenext received message.

[0060] Referring to FIG. 8A, the shaped wheel schedule determination 106is as follows. If, at 120, the VC is determined to be CBR, the schedulerprocess 52 computes 122 the earliest next time the cell can be sent bydoing the following table lookups. It performs 124 a lookup of portinformation in the ports table 66 and uses that port information todetermine the most recent time a cell (based on schedule type: must-sendor could-send or unshaped) was scheduled for transmission on this port.It also performs a lookup 126 in the VC table 60 and uses that VC tablelookup information to determine the last time a cell was last scheduledfor transmission on this VC. The scheduler 52 inserts 128 the VC index(of the VC being scheduled) into the must-send side of a slot in theshaped wheel 74 that is as close as possible to that time. If, at 120,the scheduling request is determined to be non-CBR, the schedulerprocess 52 determines 129 if the VC is nrt-VBR. If the VC is determinednot to be nrt-VBR (that is, it is rt-VBR), the scheduler process 52determines 130 the earliest next transmit time (earliest slot time) bylooking up 132 port information in ports table 66 to get the last time acell was scheduled for transmission (based on schedule type:must-send orcould-send or unshaped) on this port and by looking up 134 in the VCtable 60 to get the last time a cell was scheduled for transmission onthis VC, and finds 136 the latest time (using the traffic contractparameters) the next cell on this VC can be sent. The scheduler process52 performs 138 a search of the shaped wheel 74 to find an open slot,starting with the earliest slot time. This may be accomplished by usinghierarchical bit vectors of slots available information as a way ofsearching for an available slot to schedule. The scheduler process 52determines 140 whether or not the slot found is near the latest time(that is, whether or not the slot conforms to SCR). If it is, theprocess 52 places 142 the VC index in the must-send field in thatearliest open slot in the shaped wheel 74. If it is not, the process 52places 144 the VC-index in the could-send field instead.

[0061] If, at 129, the VC is determined to be nrt-VBR, the schedulerprocess 52 performs 130, 136 and 138 (indicated collectively byreference numeral 145) as described above. The scheduler process 52determines 146 if the slot found (at 145) is near the latest next celltransmit time (that is, if the slot conforms to SCR). If so, thescheduler process 52 places 147 the VC index in the must-send field inthat earliest open slot of the shaped wheel 74. Otherwise, the schedulerprocess 52 handles the VC under unshaped traffic scheduling (discussedbelow with reference to FIG. 8B).

[0062] Referring to FIG. 8B, the schedule determination 116 is asfollows. For a UBR scheduling request, an nrt-VBR scheduling requestthat could not be handled under shaped traffic scheduling or a firstchance queue scheduling request, the scheduler process 52 determines 150the earliest slot time this port could be transmitted to relative to thecurrent time-slot being processed on the unshaped wheel 76 by performing152 a lookup of port information in the ports table 66 to find the lasttime a cell was schedule for transmission (based on schedule type:must-send or could-send or unshaped) on this port, and adding 154 tothat last time slot an offset corresponding to a minimum inter-cell gap(which is the minimum number of slot times between transmits to thisport). It reads 155 the next time-slot a shaped VC (CBR or rt-VBR) isscheduled on the shaped wheel 74. If it determines 156 that the earliestslot time calculated above is in conflict with the next shaped VC slot,the scheduler process 52 determines 157 a slot corresponding to the nextshaped VC time-slot plus the offset. The process 52 performs 158 a rangesearch in the unshaped wheel 76 to find the earliest open slot afterthat time (that is, the shaped VC time-slot plus the offset), and enters159 the VC index in that earliest open slot in the unshaped wheel 76.Otherwise, after 158, the process 52 enters 160 the VC index in theearliest time slot (the slot corresponding to the earliest cell transmittime).

[0063]FIG. 9 illustrates the timing relationship between the shapedwheel 74 and the unshaped wheel 75, and scheduling of the shaped wheel74 for a simple example involving two CBR VCs. In the example, it isassumed that the maximum port transmit rate of a destination port “i”requires that one in every 256 slots be transmitted (or, in differentterms, requires a minimum inter-cell spacing of 255 slots). A PCR of afirst CBR VC (CBR1) to the destination port “i” requires that CBR1transmits every 500 slots. The PCR of a second CBR VC (CBR2) requiresthat that VC transmits every 1000 slots. However, the scheduling of bothof these shaped VCs must honor the port transmit rate. In the example,it is further assumed that a first CBR1 cell transmit is scheduled forslot 0 (indicated by reference numeral “161”) and a second one for slot500 (indicated by reference number “162”) at time t₀. Also assumed isthat a first cell transmit on CBR2 is scheduled for slot 1000 (referencenumber “163”) at time t₁. The spacing between the schedules for any ofthese cell transmits is greater than the minimum port inter-cell spacing(255 slots) and is therefore allowable. Now consider that a new cell isto be scheduled for CBR1 at time t₂. That cell transmit cannot occupyslot 1001 because it would violate the required port rate inter-cellspacing. Thus, the earliest next slot that can be occupied by CB1 isslot 1256 (indicated by the reference number “164”).

[0064] During scheduling on the unshaped wheel 75, the schedulerdetermines a slot 165 pointed to by the current slot pointer or index(pointer 166) on the shaped wheel and adds to that slot number at leastan offset based on the minimum inter-cell gap or spacing, which is 255in the running example, or offset 167.

[0065]FIG. 10 illustrates an exemplary port table 66 for port “i” of therunning example. As discussed earlier, the port table 66 includesentries 168 for each schedule type, entry 168 a for the must-sendschedule type, entry 168 b for the could-send schedule type and entry168 c for the unshaped schedule type. Each entry includes a field 169 afor indicating the last VC/cell transmission scheduled on the port and asecond field 169 b for indicating the port inter-cell gap. The scheduler52 reads the entry of the appropriate schedule type for this VC todetermine the last VC/cell transmission scheduled on this port andcomputes the next available slot based on the contents of the field 169a of that entry, the previous slot scheduled for this VC and the portinter-cell gap value. Because the port inter-cell gap value is the samefor each of the entries, it could be stored in a single location.Storing it in each of the entries as illustrated, however, minimizes thenumber of reads performed by the scheduler 52. Thus, when looking toschedule CBR1 at t₂, the scheduler 52 determines that, based solely onthe PCR, the next slot should be 1000, that the last cell scheduled forthis port was scheduled for slot 1000 and, given the inter-cell spacingrequired by the port transmit rate, the next earliest slot is slot 1256.

[0066] The shaping process 54 selects VCs for transmission. It iteratesover slots in the wheels, examines wheel slots/entries, and determineshow the slots will be used to transmit data.

[0067] Referring to FIG. 11, the shaper/transmit process 54 reads 170the shaped wheel 74 for the next slot to be sent. The process 54determines 172 if the next slot entry is empty. If the process 54determines 174 that the slot entry is not empty, the process determines178 if a VC index is present in the must-send field. If the processdetermines, at 178, that a VC index is present in the could-send fieldonly, the process 54 issues 180 a transmit command. If, at 178, it isdetermined that a VC index is present in the must-send field and it isfurther determined, at 182, that no VC index is present in thecould-send field, the process 54 issues a transmit command 180 totransmit the corresponding cell. If, however, at 182, the process 54determines that a must-send and could-send VC index are both present inthat slot, the process 54 sends a message to the scheduler 52 tore-schedule the could-send VC index. The process 54 then issues 180 atransmit command to transmit the corresponding cell associated with themust-send VC. The process 54 determines 186 if port flow control isasserted. If it is, the process 54 moves 187 the VC index to the firstchance queue. After the process 54 has moved the VC index to the firstchance queue, or if (at 186) port flow control is determined to bede-asserted, the process 54 sends 188 a message to the scheduler (viathe message queue) to schedule a next cell on that port (whoseflow-control was de-asserted).

[0068] If, at 172, the process 54 determines that the next shaped wheelslot entry is empty, the process 54 reads 190 the next unshaped wheel 76slot. If the process 54 determines 192 that the slot contains no validschedule information, the process 54 issues 193 a transmit skip. If avalid schedule (VC index) is present, the process 54 issues 194 that VCfor transmit and proceeds to 188.

[0069] Thus, the unshaped wheel 74 advances only when it has a cell totransmit. The shaped wheel 74 advances in all cases (that is, whether ornot it has a cell to transmit).

[0070] The shaper/transmit process 54 reconverts the scheduled slot timeto an actual hardware cycle time so that the cell is transmitted asclose to the desired time (traffic management contracted time) aspossible.

[0071] As discussed earlier, the scheduling/shaping algorithms can beimplemented in a multiprocessor, multi-threaded architecture such as theIntel® IXP™ 1200. In one such embodiment, for example, and as shown inFIG. 12, microengines (ME) 200 a and 200 b each perform TM4.1 functions.Each ME 200 supports four threads, which are used to execute twoschedulers and two shaper-transmit functions. That is, ME 200 a includesschedulers 202 a and 202 b, and ME 200 b includes schedulers 202 c and202 d. The ME 200 a includes shaper-transmit functions 204 a and 204 b,and ME 200 b includes shaper-transmit functions 206 a and 206 b.

[0072]FIG. 13 shows a third ME 200 c dedicated to the downstream receiveprocess 50 (2 threads) and the two MEs 200 a, 200 b for the downstreamTM4.1 scheduling, shaping and transmit functions (4 threads each).Scheduler and shaper-transmit threads are paired, for example, and asshown, thread pairs 210 a, 210 b, 210 c and 210 d. Such pairs 210operate on a distinct shaped and unshaped wheel. For example, pair 210 aoperates on shaped wheel 212 a and 214 a, pair 210 b operates on shapedwheel 212 b and 214 b, and so forth. This partitioning of the workloadremoves the need for mutual exclusion that would have been necessary ifall the threads dedicated to perform scheduling/shaping/transmittingwere not bound to process a sub-set of all DSL ports in the system. Inthis scheme, the ports are uniquely mapped to one of the four wheels,which means that all shaped or unshaped VCs belonging to a given portare assigned to one of shaped wheels [00-11]/unshaped wheels [00-11]. Inthis embodiment, there are four message queues (represented by arrows216 a-216 d). They are mapped on a one-to-one basis from themessage-producing receive threads to the message-consuming schedulerthreads.

[0073] As discussed above, the scheduling of the SAR 34 is performed onan as-needed basis. The must-send/could-send slot scheme allows a CBR VCto be scheduled after a VBR VC on the shaped wheel, with the CBR takingpriority. It also minimizes additional reads should the VBR schedule bestored in a separate wheel. The first chance queue allows a displacedVBR scheduled VC to be sent at the earliest opportunity for a port. Thescheduling of unshaped VCs in a relative time wheel eliminates searchingfor a port that has data, and assures the port rate is not exceeded bymaintaining that back-to-back cells are separated by an inter-cellminimum gap.

[0074] In another embodiment, as will be described with reference toFIGS. 14 through 18, the SAR device could maintain, and, consequently,the scheduling and shaping mechanisms could be implemented to use, asingle VC control structure to manage schedules for both shaped andunshaped traffic. Such a structure would therefore replace the two VCcontrol structures, that is, the VC control structure 74 for shapedtraffic and the VC control structure 75 for unshaped traffic, discussedso far.

[0075] Referring to FIG. 14, a SAR that supports a single wheel,indicated as SAR 34′, is shown. The SAR 34′, like the SAR 34 of FIG. 4described earlier, includes processes and control structures for aconcurrent shaped and unshaped traffic scheduling system. The processesinclude the receive process 50, a scheduler process 52′ and ashaper/transmit process 54′. The control structures include the receivecontrol structures 56, the shaper/transmit control structures 58 and ascheduler control structure 59′.

[0076] The scheduler control structures 59′ include the schedulermessage queue 72, as well as a VC control structure 220. In theillustrated embodiment, the VC control structure 220 is implemented as acalendar queue (“wheel”).

[0077] The VC wheel 220 hold current schedules. More specifically, theVC wheel 220 is used to schedule both shaped and unshaped traffic. Thewheel 220 is implemented as an array of time slots, including slots 222,and the slots 222 correspond to the maximum transmit rate of the networkprocessor. Each slot represents a time slot in which an ATM cell can betransmitted. The number of slots in the wheel is a function of aggregateport bandwidth. Cell transmit rate refers to the spacing between celltransmits to the network by the shaper/transmit process 54′. The wheel220 is operated in absolute time. It steps every n cycles regardless ofwhether there are any cells to be transmitted.

[0078] A VC index associated with a virtual connection is stored in aparticular one of the slots 222 during scheduling. The VC may have atraffic parameter that requires it to be serviced at least at apredetermined rate. For example, a CBR VC must conform to a PCRspecified for that VC. An rt-VBR must conform to PCR, SCR and MaximumBurst Size (MBS) parameters specified for that VC. Both CBR and rt-VBRwill need to conform to the maximum cell transfer delay (maxCTD) as ameasure of service quality. Cell Delay Variable T (CDVT) is used as ameasure of time to compensate for jitter introduced due to thescheduling inefficiency.

[0079] The VC is scheduled into the wheel 220 with sufficient spacingfor transmission with sufficient frequency to ensure that thetransmission rate conforms to both the VC cell transmit rate and theport transmit rate.

[0080] The slots 222 of the wheel 220 each reference VCs from two gradesof service rates for shaped traffic, as well as unshaped VCs. FIG. 15illustrates the format of the slot 222, which includes a must-send field224 corresponding to a first service rate (“must send” service rate)grade and a could-send field 226 corresponding to a second service rate(“could send” service rate) grade. These fields are equivalent to 78, 79in FIG. 5 that reference virtual circuits characterized by the “mustsend” and “could send” grades of service rate, respectively. A“must-send” grade indicates VCs satisfying PCR for CBR VCs or SCR forVBR VCs. The “could-send” grade indicates VCs being opportunistic as inthe case of rt-VBR VCs satisfying PCR, but of a lower priority thanmust-send. A third field, unshaped (or “Best Effort”) field 228, is usedto schedule the unshaped traffic. When the “Best Effort” field 228 andat least one of the other two fields 224, 226 store schedulinginformation, the scheduling information in the Best Effort field 228results in a re-schedule message being sent by the shaper 54′ to thescheduler 52′, as will be described. Thus, virtual circuits are selectedfor transmission from the Best Effort field 228 when must-send andcould-send virtual VCs, which have a higher priority, have not beenscheduled in the same slot. Thus, the Best Effort field 228 providesrate control for VCs that are prioritized behind VCs referenced bymust-send 224 and could-send fields 226 in a given slot.

[0081] In one implementation, the slot is organized as three longwords(32 bits). Two longwords are used to hold the contents of fields 224 and226 respectively. The other longword is used to hold the Best Effortinformation of the Best Effort field 228. Each longword has thecapability to store the port number associated with the scheduled VC ina port number information field 229. If more than one of the fields 224,226, and 228 are populated in a given slot and the VC selected fortransmission is determined to be flow controlled, the port numbers couldenable the shaper process 54′ to determine in an efficient manner thatanother VC can be selected for transmission if the port numbers indicatethat that VC and the originally selected VC are associated withdifferent ports.

[0082] Returning to FIG. 14, the port tables 66 each contain informationused by the scheduler 52 in determining schedules. Each port table 66contains entries for a given port. The port table 66 is the same as wasdescribed earlier with reference to FIGS. 4 and 10.

[0083] The shaping/transmit process 54′ is a process that manages thecontention of multiple VCs, having various service rates, fortransmission. The shaping/transmit process 54 iterates over the slots ofthe wheel 220 to select VCs for transmission. In general, theshaping/transmit process 54′ determines how time slots 222 and fieldswithin the time slots 222 will be used to transmit cells.

[0084] Further details of the wheel 220 will be provided in thedescription of the operation of the SAR downstream processes 52′ and 54′to follow.

[0085] Referring to FIG. 16, the scheduler process 52′ retrieves 100from the message queue 72 a next scheduling request in the message queue72. The scheduler process 52′ determines 102 if the request is for a“shaped” VC, that is, for a VC having a traffic parameter that requirestraffic shaping, such as CBR or rt-VBR (or even nrt-VBR, in order tomeet SCR contract). If so, the scheduler process 52′ clears 104 theprevious schedule if the message originated from the shaper 54′. If theVC is shaped, the scheduler process 52′ reads 106′ traffic parameters(from the VC table 60) and computes a schedule for a next cell on the VCin the wheel 220. The scheduler 52′ sets 108′ the schedule in the wheel220. The scheduler 52′ updates 109 the appropriate port table and VCtable with the most recent scheduling information and returns to 100 toprocess the next received message.

[0086] If, at 102, the VC is determined to be an unshaped VC, and thescheduler 52′ determines 110 that the first chance queue is not empty,the scheduler 52′ de-queues 112 an entry from the first chance queue.If, at 110, the scheduler determines that the first chance queue isempty, the scheduler determines an unshaped VC to be processed based ona WRR weight 114. The first chance queue is checked on packet boundaries(as opposed to cell boundaries).

[0087] The scheduler determines 116′ the schedule for the next cell onthis VC in the wheel 220. The scheduler 52′ sets 118′ the schedule inthe wheel 220 by storing the VC index in the appropriate slot.

[0088] Referring to FIG. 17A, the shaped schedule determination 106′ ismuch the same as described earlier with respect to FIG. 8A. Because asingle wheel is used, however, a “N” condition at decision 146 resultsin the VC index being placed in the unshaped field of the wheel 220(block 230), instead of being handled under the unshaped scheduling ofthe separate unshaped wheel of the two wheel implementation (asillustrated in FIG. 8A).

[0089] Referring to FIG. 17B, the unshaped schedule determination 116′is as follows. For a UBR scheduling request, an nrt-VBR schedulingrequest that could not be handled under shaped traffic scheduling or afirst chance queue scheduling request, the scheduler process 52′determines 150 the earliest slot time this port could be transmitted byperforming 152 a lookup of port information in the ports table 66 tofind the last time a cell was transmitted on this port, and adding 154to that last time slot an offset corresponding to a minimum inter-cellgap (which is the minimum number of slot times between transmits to thisport). The scheduler process 52′ determines 157 a slot corresponding tothe next unshaped VC time-slot plus the offset. The process 52′ performs158 a range search in the wheel 220 to find the earliest open slot afterthat time (that is, the unshaped VC time-slot plus the offset), andplaces 240 the VC index in the unshaped field of that earliest open slotin the wheel 220.

[0090] Referring to FIG. 18, the shaper/transmit process 54′ reads 270the wheel 220 for the next slot 222 to be sent. The process 54′determines 276 if a VC index is present in the must-send field. If, at276, it is determined that a VC index is present in the must-send fieldand it is further determined, at 278, that no VC index is present in thecould-send field, and, at 280, that no VC index is present in the BestEffort field, the process 54′ determines 282 if port flow control isasserted. If it is, the process 54′ moves 284 the VC index to the firstchance queue. After the process 54′ has moved the VC index to the firstchance queue, it goes to sleep 286 for the cell transmit time. If, at282, it is determined that port flow control is not asserted, theprocess 54′ issues 287 a transmit command to transmit the correspondingcell and sends 288 a message to the scheduler 52′ to scheduler the nextcell. After 287 or 288, the process 54′ returns to 270 to read the nextwheel entry.

[0091] If, at 278 and 280, the process 54′ determines that either acould-send VC index or Best Effort VC index, or both, is present in thatslot along with a must-send VC index, the process 54′ sends 290 are-schedule message to the scheduler 52′ prior to determining if portcontrol is asserted at 282. The re-schedule message informs thescheduler 52′ that the VC corresponding to the VC index in the BestEffort and/or could-send fields is to be re-scheduled.

[0092] If, at 276, the process 54′ determines that the must-send fieldis empty, the process 54′ determines 292 if a could-send VC index ispresent in the entry. If so, and the process 54′ further determines 294that a Best Effort VC index is also present in the entry, the process54′ sends 290 a re-schedule message to the scheduler to re-schedule theBest Effort VC and proceeds to 282. Otherwise, if no Best Effort VC isscheduled in the entry, the process 54′ does not send a re-schedulemessage but proceeds directly to 282. If, at 292, the process 54′determines that no could-send VC index is present in the entry, theprocess 54′ still checks for a Best Effort VC index in the Best Effortfield. If that field contains a VC index, the process 54′ proceeds to282. If, at 296, it is determined that no Best Effort VC is present, theprocess 54′ proceeds to 286.

[0093] Returning to the multi-threaded processing architecture of FIGS.12-13, it will be understood that a single wheel such as the wheel 220described above could be used instead of the two wheel arrangement thatis shown. Thus, each of the thread pairs 210 would operate on one wheelas opposed to two separate wheels for shaped and unshaped traffic.

[0094] Other embodiments are within the scope of the following claims.

What is claimed is:
 1. A method comprising: determining transmitschedules for first cells and second cells based on service rates andmaximum port rates; and selecting for transmission the first and secondcells based on the transmit schedules so that the transmit schedules ofthe first cells take priority over the transmit schedules for the secondcells.
 2. The method of claim 1, further comprising: associating thefirst cells with shaped virtual circuits that require traffic shaping toensure service rates specified for the shaped virtual circuits;associating the second cells with unshaped virtual circuits that do notrequire traffic shaping to ensure service rates specified for theunshaped virtual circuits; and associating the first and second cellswith ports from which such cells are to be transmitted, the ports havingmaximum port rates.
 3. The method of claim 2, further comprising:providing a shaped control structure for maintaining transmit schedulesfor the first cells; and providing an unshaped control structure formaintaining transmit schedules for the second cells.
 4. The method ofclaim 3, wherein the shaped and unshapedcontrol structures arepartitioned into time slots, the number of time slots based on a maximumaggregate port bandwidth.
 5. The method of claim 2, wherein the servicerates specified for the shaped virtual circuits comprise a constant bitrate (CBR).
 6. The method of claim 5, wherein the service ratesspecified for the shaped virtual circuits comprise a real-time variablebit rate (rt-VBR).
 7. The method of claim 4, wherein determiningcomprises: for each first cell, by order of scheduling request,computing an earliest next cell transmit time.
 8. The method of claim 7,wherein the slots in the shaped control structures comprise a must sendfield and a could send field, and the service rates specified for theshaped virtual circuits comprise a constant bit rate (CBR) and a servicerate that is not CBR.
 9. The method of claim 8, wherein determiningfurther comprises: determining if the associated service rate is CBR; ifthe associated service rate is determined to be CBR, determining fromamong the slots in the shaped control structure an earliest open timeslot closest in time to the earliest next cell transmit time; andentering into the must send field of the earliest open time slot avirtual circuit index for the virtual circuit with which such first cellis associated.
 10. The method of claim 9, wherein if the associatedservice rate is determined not to be CBR, the method further comprises:determining a latest next cell transmit time; searching the slots in theshaped control structure for an earliest open time slot, beginning withthe a slot closest in time to the earliest next cell transmit time;determining if the earliest open time slot is near in time to the latestcell transmit time; if the earliest open time slot is determined to benear in time to the latest cell transmit time, entering into the mustsend field of the earliest time slot a virtual circuit index for thevirtual circuit with which such first cell is associated; and otherwise,if the associated service rate is rt-VBR, entering the virtual circuitindex into the could-send field.
 11. The method of claim 10, furthercomprising: if the earliest open time slot is determined not to be nearin time to the latest cell transmit time and the associated service rateis nrt-VBR, handling the first cell as a second cell for the purpose oftransmit schedule determination.
 12. The method of claim 10, whereinsearching comprises searching a hierarchical arrangement of bit vectorsused to convey available slot information.
 13. The method of claim 10,further comprising: providing an array of entries have associatedpriorities; providing a first chance queue having an associated prioritythat is higher that the priorities associated with the entries of thearray; and associating each second cell with a selected one of theentries when a scheduling request is received for such cell by storing avirtual circuit index for the virtual circuit with which such secondcell is associated in the selected entry.
 14. The method of claim 13,wherein selecting comprises: reading a current slot in the shapedcontrol structure; if the current slot is non-empty and includes a VCindex in the must send field, issuing a transmit command to transmit acorresponding cell; if the current slot also includes a VC index in thecould send field, moving the VC index to an entry in the first chancequeue; and if the current slot in the shaped control structure is empty,issuing a transmit command to transmit a corresponding cell in a currentslot of the unshaped control structure.
 15. The method of claim 14,further comprising if port flow control is asserted for the port fromwhich the corresponding cell is to be transmitted, moving the VC indexof the current slot to the first chance queue.
 16. The method of claim14, wherein determining further comprises: if there is no schedulerequest for one of the first cells, then determining if the first chancequeue is non-empty; if the first chance queue is determined to benon-empty, dequeueing an entry from the first chance queue and copying aVC index from the entry to a slot in the unshaped control structure; andif the first chance queue is determined to be empty, reading an entryfrom the array based on a Weighted Round Robin selection and copying aVC index from the entry into a slot in the unshaped control structure.17. The method of claim 16, wherein determining further comprisesdetermining the slot in the unshaped control structure as a slot whichcorresponds to an earliest cell transmit time slot relative to thecurrent slot in the shaped control structure when such earliest celltransmit time slot is not in conflict with a next slot available forselection in the shaped control structure.
 18. The method of claim 17,wherein determining the slot in the unshaped control structure furthercomprises: determining a last time slot in which a cell was transmittedfrom a port for the associated virtual circuit; and adding to the lasttime slot an offset number of slots corresponding to a minimuminter-cell gap for the port.
 19. The method of claim 2, wherein theservice rates specified for the unshaped virtual circuits compriseunspecified bit rates (UBR).
 20. The method of claim 19, wherein theservice rates specified for the unshaped virtual circuits furthercomprise non-real-time VBR (nrt-BR).
 21. The method of claim 2, furthercomprising: providing at least one control structure to maintaintransmit schedules for both the first and second cells, the controlstructure being partitioned into time slots.
 22. The method of claim 21,wherein each slot includes a plurality of fields, the plurality offields including a must send field and a could send field for scheduletypes associated with shaped virtual circuits, and a best effort fieldfor schedule types associated with unshaped virtual circuits.
 23. Themethod of claim 22, wherein determining comprises: for each first cell,by order of scheduling request, selecting an earliest available one ofthe slots based on an earliest next cell transmit time.
 24. The methodof claim 23, further comprising: providing an array of entrieseachhaving an associated priority; providing a first chance queue having anassociated priority that is higher than the priorities associated withthe entries of the array; and associating each second cell with aselected one of the entries when a scheduling request is received forsuch cell by storing a virtual circuit index for the virtual circuitwith which such second cell is associated in the selected entry.
 25. Themethod of claim 24, wherein determining further comprises: if there isno schedule request for one of the first cells, then determining if thefirst chance queue is non-empty; if the first chance queue is determinedto be non-empty, dequeueing an entry from the first chance queue andcopying a VC index from the entry to the best effort field in anearliest available one of the slots in the control structure based on anearliest next cell transmit time; and if the first chance queue isdetermined to be empty, reading an entry from the array based on aWeighted Round Robin selection and copying a VC index from the entryinto the best effort field in an earliest available one of the slots inthe control structure based on an earliest next cell transmit time. 26.The method of claim 22, wherein selecting comprises: reading a currentslot in the control structure; if the current slot is non-empty andincludes a VC index in one of the must send and could send fields,issuing a transmit command to transmit a corresponding cell; if thecurrent slot also includes a VC index in the best effort field, issuinga command to re-schedule a corresponding cell; and if the current slotis non-empty and includes a VC index in only the best effort field,issuing a command to transmit a corresponding cell.
 27. The method ofclaim 26, further comprising if port flow control is asserted for a portfrom which the corresponding cell is to be transmitted, moving the VCindex of the current slot to the first chance queue.
 28. The method ofclaim 26, further comprising if port flow control is asserted for a portfrom which the corresponding cell is to be transmitted, determining ifanother one of the fields stores a VC index and is associated with aport that different from the port for which port flow control isasserted, issuing a command to transmit a cell corresponding to the VCindex in such field.
 29. The method of claim 21, further comprisingconverting a bandwidth value expressed in bits per second to a timeslot.
 30. The method of claim 21, wherein the at least one controlstructure comprises multiple control structures, each mapping to adifferent port, and wherein the determining and the selecting areperformed by multiple execution threads, and each pair of threadsperforming the determining and the selecting operates to process adifferent one of the control structures.
 31. The method of claim22,wherein determining comprises performing separate port bandwidthcomputations for each of the schedule types.
 32. An article comprising:a storage medium having stored thereon instructions that when executedby a machine result in the following: determining transmit schedules forfirst cells and second cells based on service rates and maximum portrates; and selecting for transmission the first and seconds cells basedon the transmit schedules so that the transmit schedules of the firstcells take priority over the transmit schedules for the second cells.33. The article of claim 32, wherein the instructions further compriseinstructions which when executed on a machine result in the following:associating the first cells with shaped virtual circuits that requiretraffic shaping to ensure service rates specified for the shaped virtualcircuits; associating the second cells with unshaped virtual circuitsthat do not require traffic shaping to ensure service rates specifiedfor the unshaped virtual circuits; and associating the first and secondcells with ports from which such cells are to be transmitted, the portshaving maximum port rates.
 34. A network data aggregation device havingfirst ports for receiving packets from a service network andtransmitting cells associated with the packets from second ports to aservice user over a DSL link, comprising: a receiving device to generaterequests to schedule the cells for transmission on virtual circuits fromthe second ports; scheduling data structures including a first datastructure associated with a first service type and a second datastructure associated with a second service type; and a scheduler toprocess the requests, the scheduler determining, for each request, aservice type and determining from the service type an associated portrate a schedule, and entering the schedule into the first data structureor the second data structure based whether the service type correspondsto the respective first service type or second service type.
 35. Thenetwork data aggregation device of claim 34, further comprising: atraffic shaper to select a cell for transmission based on the schedulesin the first and second scheduling data structures, the traffic shapergiving priority to schedules in the first scheduling data structure. 36.A network data aggregation device having first ports for receivingpackets from a service network and transmitting cells associated withthe packets from second ports to a service user over a DSL link,comprising: a receiving device to generate requests to schedule thecells for transmission on virtual circuits from the second ports; ascheduling data structure having slots with one or more fields to storea schedule for traffic associated with a first service type and one ormore fields to store a schedule for traffic associated with a secondservice type; and a scheduler to process the requests, the schedulerdetermining, for each request, a service type and determining from theservice type an associated port rate a schedule, and entering theschedule into one of the fields in one of the slots based whether theservice type corresponds to the respective first service type or secondservice type.
 37. The network data aggregation device of claim 36,further comprising: a traffic shaper to select a cell for transmissionbased on the schedules in the scheduling data structure, the trafficshaper giving priority to schedules in the one or more fields that storea schedule for traffic associated with the first service type.